What is an easy way to reproduce the above failure? cc: @maleadt

We could make it so that passing Ref(arg) actually passes by reference, and doesn't do the by-value conversion.

What does

Entry function 'ptxcall_calculate_Gu__66' uses too much parameter space (0x16c8 bytes, 0x1100 max)

mean?

CUDA uses a special buffer, the parameter space, to put arguments in. This buffer is about 4K large, and has special semantics that benefit performance (read-only, so threads can read from it without synchronizing, etc). Although arguments in Julia are normally passed by reference, i.e. putting pointers in that space, when invoking kernels we change the calling convention and pass by reference such that loading e.g. the size or pointer of an array doesn't synchronize threads. That works great, until you pass a large (number of) arguments as you apparently do.

Thanks @maleadt, that's very helpful!

In this PR, we haven't directly changed any kernel function signatures. However, this PR does pass more complicated objects into kernels (a wrapper around an OffsetArray called a "Field", rather than simply the OffsetArray). The primary changes in this PR are thus 1. not to extract the underlying OffsetArray from a Field, and 2. writing an adapt_structure method for Fields. I suppose the translation that's performed by adapt_structure increases the number or arguments to the function ptxcall_calculate_Gu__66?

The changes made in this PR are not strictly necessary --- they are a convenience. If manually unwrapping Fields (the method we previously used) is necessitated by CUDA limitations, I think we can live with that. If I understand this issue correctly, we are facing a basic trade-off between (compiler?) performance and the use of convenient but complicated abstraction objects?

If I understand this issue correctly, we are facing a basic trade-off between (compiler?) performance and the use of convenient but complicated abstraction objects?

It's a hardware limitation, really. The compiler could anticipate though, e.g. by not passing very large objects by value, or by providing an escape hatch (like the Ref suggestion in JuliaGPU/CUDA.jl#267). You can experiment with this yourself, by changing which arguments get tagged byval in https://github.com/JuliaGPU/GPUCompiler.jl/blob/master/src/irgen.jl#L607, and changing the logic that packs arguments in https://github.com/JuliaGPU/CUDA.jl/blob/master/lib/cudadrv/execution.jl#L8-L37 accordingly (to pass a pointer to a pointer instead of a pointer to a value).

Superceded by #1057